Recently, technology that can process a variety of digital content substantially in real time, such as audio, image, and video, is commonly being used. For example, in terrestrial digital TV, substantially real-time processing of data exceeding 2M pixels/frame, such as 1080i (1920 (horizontal)×1080 (vertical), 29.97 frames/second), is required. For such applications, SIMD super-parallel computational processing devices have been developed. An SIMD super-parallel computational processing device includes a plurality of programmable arithmetic processor elements (PEs), and a single computation based on a single instruction is executed concurrently by all the PEs, thereby enabling high-speed computational processing. Depending on the algorithm used, however, data preparation in advance, post-processing of the obtained computation results, and/or data transfer constitutes a bottleneck in the overall speed, and this might prohibit high-speed operation. In an example configuration, an SIMD super-parallel computational processing device includes input and output memory blocks for storing data involved in processing, and a plurality of PEs that read data from the input memory block in units of a preset number of bits, such as 1, 2, 4, or 8 bits, and that execute computational processing and write the computation results to the output memory block in units of the same number of bits. The individual PEs are configured in the form of arrays and are connected to each other via channels, which allows concurrent processing for a variety of computations. Examples of SIMD super-parallel computational processing devices that are mass-produced include MX-G from Renesas Electronics Corporation and C-ViA from Toshiba Corporation.
In the case of terrestrial digital TV, there is a problem in that each frame constituting TV video includes degraded information, such as optical blurring or unsharpness, although it is not blurred to such an extent that objects in the frame are indistinguishable.
FIG. 1 shows an example of optically degraded information included in a frame of TV video. FIG. 1 includes two images: the left image represents a frame constituted only of Y (luminance) components of TV video acquired by using an X-ray pinhole camera; on the other hand, the right image represents an image restored by subjecting the left image in FIG. 1 to super-resolution processing using related art invented by the inventor of the present invention (Patent Literatures 1 and 2). A comparison between the images in FIG. 1 indicates that TV video before the super-resolution processing includes degraded information, such as optical blurring or unsharpness.
In the image restoration technologies invented by the inventor of the present invention (Patent Literatures 1 and 2), while repeating iterations according to formulas based on Bayesian probability theory for obtaining a restored image from information about one still image including degraded information, such as optical blurring or unsharpness, a maximum-likelihood OTF (Optical Transfer Function), which is a Fourier transform product of a maximum-likelihood PSF (Point Spread Function) having a maximum likelihood for the luminance distribution of the still image, as well as a maximum-likelihood restored image, are estimated through numerical computations. However, since the image restoration technologies employ algorithms involving computations executed in real space and in the spatial frequency domain according to nonlinear equations, it is necessary to execute a Fourier transform and an inverse Fourier transform multiple times per iteration, which results in a huge amount of computational processing. Thus, there has been a problem in that it is difficult to handle TV video, which requires real-time processing. Also, data preparation in advance, post-processing of the computation results, and data transfer constitute bottlenecks in the overall speed when using an SIMD super-parallel computational processing device. Thus, there has been a problem in that the algorithms are not suitable for SIMD super-parallel computational processing devices.
The inventor of the present application has filed a patent application, which is now on file, for technology L in which the inventor improved the image restoration technologies invented by himself (Patent Literatures 1 and 2) to allow substantially real-time processing of TV video. In technology L, the inventor changed the type of numbers handled in the computations in the image restoration technologies according to the related art (Patent Literatures 1 and 2) from complex numbers to real numbers and also changed the type of computational processing device from a software implementation to a hardware implementation using FPGAs (Field Programmable Gate Arrays), allowing substantially real-time processing. However, there have been problems in that the number of gates in the hardware implementation is as large as 1.5 million gates, and thus, costs are high and the installation area is large. Also, in order to realize substantially real-time processing, the possible number of iterations is six at most, and thus there has been a concern that this might result in inadequate robustness of super-resolution processing. Furthermore, since the algorithms used are based on nonlinear equations, there has been a problem in that data preparation in advance, post-processing of the computation results, and data transfer constitute bottlenecks in the overall speed when using an SIMD super-parallel computational processing device, which makes the algorithms unsuitable for SIMD super-parallel computational processing devices.
The image restoration ability of technology L is comparable to those of the image restoration technologies (Patent Literatures 1 and 2) in the case where a frame is not blurred to such an extent that objects in the frames are indistinguishable. However, convergence is so slow that there are many cases where about six iterations are required before an image is fully restored. Thus, problems may arise in terms of the processing speed for applications involving substantially real-time processing of TV video with a large screen, such as 2K to 4K, which requires even higher operation speeds and further improvement in the convergence speed. Accordingly, in order to further increase the operation speed of technology L, the inventor of the present invention improved the algorithms employing nonlinear equations based on Bayesian probability theory used in the image restoration technologies (Patent Literatures 1 and 2), thereby inventing an accelerated algorithm that employ nonlinear equations based on Bayesian probability theory, and has filed a patent application for this accelerated algorithm. In the TV-video super-resolution processing technology based on this accelerated algorithm, the PSF restoring step used in the image restoration technologies (Patent Literatures 1 and 2) and technology L is omitted. Furthermore, a first PSF associated with a degradation index corresponding to the degree of degradation of a degraded image and a second PSF obtained by sharpening the first PSF in advance are used. This makes it possible to provide an image restoration ability comparable to that of technology L just with two iterations by using Bayes probability formulas for obtaining a restored image from information about a still image including degraded information. With the TV-video super-resolution processing technology based on the accelerated algorithm, however, the amount of computation is still large since the algorithm is based on nonlinear equations. Furthermore, there has been a problem in that data preparation in advance, post-processing of the computation results, and data transfer constitute bottlenecks in the overall speed when using an SIMD super-parallel computational processing device, which makes the algorithm unsuitable for SIMD super-parallel computational processing devices.
As an example of an already disclosed algorithm invented by other inventors and suitable for super-parallel computational processing devices, there is a method in which image blurring is removed based on equation (1) by using an MISD (Multiple Instruction stream, Single Data stream) SAP (Systolic Array Processor) in which processors are arrayed two-dimensionally (Patent Literature 3). However, since this method involves iterations based on the linear equation according to equation (1), which are processed by the MISD SAP, there has been a problem that it is not directly applicable to an SIMD super-parallel computational processing device.[Eq. 1]U(n+1)=U(n)−k*(H*U(n)−Yb)−S*U(n)  (1)
In equation (1), U signifies an original image, H signifies a PSF, Yb signifies a degraded image, k signifies a feedback operator, and S signifies a smoothing operator. These are all two-dimensional functions constituted of m×n elements. * signifies a two-dimensional convolution operator.
As other super-resolution algorithms based on linear equations, for example, in a method and device disclosed in Patent Literature 4, a state space model (a state equation constituted of an original image and an observation equation constituted of the original image and noise) is configured without estimating autoregression parameters of a degraded image, and image restoration is executed by using a Kalman filter on the state space model. With this method and device, however, there is a problem in that what is restored is only a region in the vicinity of a pixel of interest, not an entire image, and there is a possibility that the method and device were not developed for SIMD super-parallel computational processing devices.